Update: re-harvest qwen3_14b_decode and scale it to all 40 layers - #1506
Merged
ChaoWao merged 1 commit intoJul 27, 2026
Merged
Conversation
|
Important Review skippedToo many files! This PR contains 108 files, which is 8 over the limit of 100. To get a review, narrow the scope: Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (123)
You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Collaborator
Author
|
CI triage: everything is green except It is Will rebase once #1504 lands. Everything else on this run passes, including |
ChaoWao
force-pushed
the
fix/issue-1484-qwen3-decode-reharvest
branch
3 times, most recently
from
July 27, 2026 01:59
076aacc to
218f57b
Compare
Fixes hw-native-sys#1484 The example carried pypto codegen last refreshed on 2026-06-25, cited a provenance file upstream had deleted (`models/qwen3/14b/decode_layer.py`), and covered 2 of Qwen3-14B's 40 layers. Re-harvested from `decode_fwd.py`'s `decode_fwd_layers` at pypto-lib `45be52c` with `_CHUNK_NLAYERS = 40`, and the README now records that commit plus the ptoas and pto-isa pins so the next drift is detectable rather than archaeological. Scaling to the full model is nearly free. The generated orchestration loops over layers (`for (int64_t i = 0; i < 40; i += 1)`) instead of unrolling them, so going from 2 to 40 moves a trip count and leaves the kernel set the same size. The lib chunks its dispatches because a 40-layer one used to exceed a 2000 ms stream-sync timeout; simpler raised that to 50 s in hw-native-sys#1175 and a 40-layer step is ~40 ms, so the constraint is gone. What did change is attention. Since pypto-lib hw-native-sys#765 and hw-native-sys#796 it is a CANN FusedInferAttentionScore extern rather than generated kernels, folding per-head QK-norm, RoPE, the paged KV write, the flash inner loop and the online softmax into one mixed AIC+2xAIV task behind an FFTS barrier. Seven kernels go away (`rope_qkv`, `qk_gamma`, `qk_recip`, `fa_work_build`, `fa_fused_aic`/`_aiv`, `online_softmax`, `attn_fence`) and the harvested extern arrives under `kernels/vendor/paged_attention_cce/`. That extern needs CANN devkit headers, which nothing could express: every incore compiled against the same fixed include set, even though `KernelCompiler.compile_incore` has always taken `extra_include_dirs`. The scene-test `CALLABLE` now carries a per-incore `extra_include_dirs`, resolved when the kernel is compiled rather than when its module is imported — sim and macOS runners collect a case declaring a CANN dependency without having CANN. Entries may use `$VAR` and may name directories that do not exist here, so one candidate list spans SDK layouts; an unset variable or a set where nothing survived raises, which beats a "file not found" from four includes deep in vendored code. The harvested tree is copied verbatim and must stay that way: `fai_body.hpp` reaches its dependencies by relative include, and reformatting it would make the next refresh diff against our copy instead of against upstream — the exact drift this example exists to expose. It therefore lives under `kernels/vendor/`, and the repo's header, formatting and language lint skip anything below a `vendor/` directory. Keying the carve-out on the directory rather than on this operator's name means the next harvested extern needs no lint change at all. The extern also moves the paged KV layout from NSND to vLLM's BSND (`[page, token, kv_head, dim]`, so `slot_mapping` is directly the row index), which the golden now models for both the attention read and the KV write-back. The previous harvest needed one hand-edit, replacing a `[[block_local]] static` in `fa_fused_aiv` that the AICore loader rejected. That kernel no longer exists and the current codegen emits no such construct, so this harvest is unmodified codegen throughout. At 40 layers the case takes ~5 min against a 38.05 GiB fixture (24.61 weights + 13.44 paged KV, bf16) — over half the onboard sweep's 600 s session budget. It runs as its own CI step on a dedicated device instead, keeping that sweep's short timeout meaningful as a hang detector. Verification: passes on device with output and all 40 layers' KV caches matching the torch reference at RTOL=5e-2 / ATOL=1e-1, with no ring-sizing overrides — per-layer intermediates live inside that iteration's scope, so the live set does not grow with layer count. The l0-swimlane docs used this example as their "real SPMD workload", shrinking the generated fa_fused via `--set-arg 0=96` on its work-item count. The extern takes its work distribution from runtime tiling metadata instead, so no `--set-arg` shrinks it and those references are removed rather than repointed. The dfx buffer-capacity row measured at 2 layers is dropped for the same reason: it would now be a stale number rather than a wrong one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ChaoWao
force-pushed
the
fix/issue-1484-qwen3-decode-reharvest
branch
from
July 27, 2026 02:39
218f57b to
370d108
Compare
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 27, 2026
… dots `fix-pr` Step 6 told you to squash first and rebase after: the `> 1` row soft-resets to `$BASE_REF`, and only the following list says "rebase onto `$BASE_REF`". `commit-and-push` has the opposite (correct) order, so the two documents disagreed and the skill's order is the one that corrupts the commit. `git reset --soft "$BASE_REF"` moves HEAD to the base but keeps the branch's index. If the branch still sits on an older base — which it does whenever anything merged while the PR was open — every file the base gained meanwhile is recorded as a deletion by the PR. Rebasing afterwards does not repair it: the revert is by then part of the PR's own diff and replays cleanly onto the new base. It surfaces as unrelated files being reverted, easy to miss in a large diff and easy to merge. Hit while squashing hw-native-sys#1506: `docs/troubleshooting/device-error-codes.md` came back as -86 lines because hw-native-sys#1501 had merged since the branch point. Step 6 now fetches and rebases at the top, *before* the edit step rather than after it: git refuses to rebase a dirty worktree, so a rebase placed after the fix cannot run at all. Verification gains a three-dot diff of the file list, and it lives with the single-commit check after the commit exists — at the soft-reset itself `HEAD` is `$BASE_REF`, so that diff is empty by construction and would have verified nothing. Two dots would also list files the base has and the branch does not, rendering them as the branch's deletions: it both hides a real revert in the noise and invents fake ones. The single-commit check now separates its outcomes, since the surrounding table already treats them differently: `> 1` squashes again, `0` stops. `commit-and-push` §1 gains the fetch, and its soft-reset step states the precondition it silently relied on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ChaoWao
added a commit
that referenced
this pull request
Jul 27, 2026
… dots (#1516) `fix-pr` Step 6 told you to squash first and rebase after: the `> 1` row soft-resets to `$BASE_REF`, and only the following list says "rebase onto `$BASE_REF`". `commit-and-push` has the opposite (correct) order, so the two documents disagreed and the skill's order is the one that corrupts the commit. `git reset --soft "$BASE_REF"` moves HEAD to the base but keeps the branch's index. If the branch still sits on an older base — which it does whenever anything merged while the PR was open — every file the base gained meanwhile is recorded as a deletion by the PR. Rebasing afterwards does not repair it: the revert is by then part of the PR's own diff and replays cleanly onto the new base. It surfaces as unrelated files being reverted, easy to miss in a large diff and easy to merge. Hit while squashing #1506: `docs/troubleshooting/device-error-codes.md` came back as -86 lines because #1501 had merged since the branch point. Step 6 now fetches and rebases at the top, *before* the edit step rather than after it: git refuses to rebase a dirty worktree, so a rebase placed after the fix cannot run at all. Verification gains a three-dot diff of the file list, and it lives with the single-commit check after the commit exists — at the soft-reset itself `HEAD` is `$BASE_REF`, so that diff is empty by construction and would have verified nothing. Two dots would also list files the base has and the branch does not, rendering them as the branch's deletions: it both hides a real revert in the noise and invents fake ones. The single-commit check now separates its outcomes, since the surrounding table already treats them differently: `> 1` squashes again, `0` stops. `commit-and-push` §1 gains the fetch, and its soft-reset step states the precondition it silently relied on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1484.
What
Re-harvests
examples/a2a3/tensormap_and_ringbuffer/qwen3_14b_decode/frompypto-lib
45be52cand scales it from a 2-layer chunk to all 40 Qwen3-14Bdecode layers in one dispatch.
Two commits: the
CALLABLEinclude-dirs hook the harvest needs, then theharvest itself.
Settling the issue's open question
The issue asked whether to run 40 layers as one fused chunk or as 5 dispatches
of the upstream chunk-8, and recommended the latter to keep codegen size down.
That reason does not hold. The generated orchestration loops over layers —
for (int64_t i = 0; i < 40; i += 1)— so going 2 → 40 moves a trip count andleaves the kernel set the same size. One dispatch it is.
The lib chunks because a 40-layer dispatch used to exceed a 2000 ms stream-sync
timeout (
decode_fwd.py:1153). simpler raised that to 50 s in #1175 and a40-layer step is ~40 ms, so the constraint is gone.
Two other framings in the issue turned out not to apply:
@scene_test(level=)is the runtime hierarchy level (L2 chip vs L3 worker), not a CI cost tier, and
layer count is baked into the orchestration
.cpp, so it cannot be a per-CASESparam — one layer count per class.
What actually changed: attention is no longer codegen
Since pypto-lib #765/#796 the attention stage is a CANN
FusedInferAttentionScoreextern, not generated kernels. It folds per-headQK-norm, RoPE, the paged KV write, the flash inner loop and the online softmax
into one mixed AIC+2×AIV task behind an FFTS barrier.
rope_qkv,qk_gamma,qk_recip,fa_work_build,fa_fused_aic/_aiv,online_softmax,attn_fence.kernels/paged_attention_cce/, copied verbatim.kernel/fai_body.hppreaches its dependencies by relative include, sosplitting the tree would mean patching the source and re-patching on every
refresh. Its
vendor/subtree is CANN source under Huawei copyright(~16k LOC). Verbatim also means our header/format/language lint must leave it
alone — hence the carve-out in
.pre-commit-config.yamlandtests/lint/check_headers.py.[page, token, kv_head, dim], soslot_mappingis directly the row index). The golden models the new layout onboth the attention read and the KV write-back.
CALLABLE["incores"][i]["extra_include_dirs"]is for.$ASCEND_HOME_PATHkeeps them off machine-specific paths.
This depends on #1497 — AscendC's block-local globals carry relocations no
amount of inlining removes, so these kernels could not load before it.
Issue item 3: the hand-edit is gone
The previous harvest patched
fa_fused_aivto replace a[[block_local]] staticthe loader rejected. That kernel no longer exists and the currentcodegen emits no such construct, so this harvest is unmodified codegen
throughout — a refresh is now a plain copy.
Verification
reference at
RTOL=5e-2 / ATOL=1e-1. ~5 min wall on an idle die.that iteration's scope, so the live set does not grow with layer count — the
default heap / task window / dep pool carry the 40-layer graph.
HBM, matching the issue's estimate.
Python UTs: 829 passed, 2 skipped. Full pre-commit clean on the diff.
Docs
README records the exact pypto-lib commit plus the ptoas/pto-isa pins and a
regeneration recipe that works (
codegen_only=True, no device). Also updated:multi-repo-qwen-setupPath 0,docs/testing.mdfor the new key.The l0-swimlane docs used this example as their "real SPMD workload", shrinking
the generated
fa_fusedvia--set-arg 0=96. The extern takes its workdistribution from runtime tiling metadata, so no
--set-argshrinks it — thosereferences are removed rather than repointed at something unverified. The
dfx buffer-capacity row measured at 2 layers is dropped for the same reason.
Notes for review
kernel_config.py: it marksk_cache/v_cacheas plain inputs, but the extern writes the current token's KV into them. The
CALLABLEmarks themINOUTso simpler copies the pools back and the goldencan verify all 40 layers' KV writes — the stronger check the issue asks for.
hand-written surface is the test file, the golden, and the lint carve-outs.
🤖 Generated with Claude Code